The French MEDIA/EVALDA Project: the Evaluation of the Understanding Capability of Spoken Language Dialogue Systems
نویسندگان
چکیده
This paper presents and reports on the progress of the EVALDA/MEDIA project, focusing on the recording and annotating protocol of the reference dialogue corpus. The aim of this project is to design and test an evaluation methodology to compare and diagnose the context-dependent and independent understanding capability of spoken language dialogue systems. Systems from both academic organisations (IRIT, LIA, LIMSI, LORIA, VALORIA, CLIPS) and industrial sites (FRANCE TELECOM R&D, TELIP) will be tested. ELDA is the coordinator of the Technolangue/EVALDA multi-campaign evaluation project, of which MEDIA is a subcampaign. LIMSI is the scientific coordinator of the project. MEDIA began in January 2003.
منابع مشابه
A Deep-Parsing Approach to Natural Language Understanding in Dialogue System: Results of a Corpus-Based Evaluation
This paper presents an approach to dialogue understanding based on a deep parsing and rule-based semantic analysis. Its performance in the semantic evaluation performed in the framework of the EVALDA/MEDIA campaign is encouraging. The MEDIA project aims to evaluate natural language understanding systems for French on a hotel reservation task (Devillers et al., 2004). For the evaluation, five pa...
متن کاملResults of the French Evalda-Media evaluation campaign for literal understanding
The aim of the MEDIA-EVALDA project is to evaluate the understanding capabilities of dialog systems. This paper presents the MEDIA protocol for speech understanding evaluation and describes the results of the June 2005 literal evaluation campaign. Five systems, both symbolic or corpus-based participated to the evaluation which is based on a common semantic representation. Different scorings hav...
متن کاملOverview of the ARISE project
The LE-3 project ARISE (Automatic Railway Information Sys tems for Europe) ran from October 1996 through December 1998. Four prototypes of train timetable information systems for three different languages were developed, tested, and validated: Italian (based on technology developed by CSELT), French (two systems, one based on Philips technology, the other on technology by LIMSI), and Dutch (ba...
متن کاملThe PEACE SLDS understanding evaluation paradigm of the French MEDIA campaign
This paper presents a paradigm for evaluating the context-sensitive understanding capability of any spoken language dialog system: PEACE (French acronym for Paradigme d’Evaluation Automatique de la Compréhension hors et En-contexte). This paradigm will be the basis of the French Technolangue MEDIA project, in which dialog systems from various academic and industrial sites will be tested in an e...
متن کاملLeveraging study of robustness and portability of spoken language understanding systems across languages and domains: the PORTMEDIA corpora
The PORTMEDIA project is intended to develop new corpora for the evaluation of spoken language understanding systems. The newly collected data are in the field of human-machine dialogue systems for tourist information in French in line with the MEDIA corpus. Transcriptions and semantic annotations, obtained by low-cost procedures, are provided to allow a thorough evaluation of the systems’ capa...
متن کامل